Skip to content

fix(chat): preserve image inputs across conversation engines - #1164

Open
Robert0622 wants to merge 3 commits into
mainfrom
codex/chat-image-input
Open

Robert0622 wants to merge 3 commits into
mainfrom
codex/chat-image-input

Conversation

@Robert0622

@Robert0622 Robert0622 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

Chat staged uploaded images as files and gave every engine the paths as text. Visual input depended on each engine's ability to open the file through a tool.

  • Send current-turn images as native inputs to CodeBuddy, Codex, Claude Code, OpenCode and Pi.
  • Expose workbench_read_image as a read-only session MCP tool for engines using tool-based image input and for historical attachments. Validate the attachment ID against the turn's authorized attachments and return standard MCP image content.
  • Bound runner stdin writes for large image messages while preserving UTF-8 and path remapping. Native session handoffs reuse images already recorded in the journal.
  • Advertise image input in Kimi and DeepSeek custom model configurations when the session has an authorized image. This preserves MCP image results in upstream model requests.

Scope

  • Platform core / API / runner

Compatibility

Non-image attachments continue to use staged paths. Text-only turns use the existing prompt path. OpenCode, Pi, Kimi Code and DeepSeek Harness use runner-side model configuration for multimodal input. Remote execution needs the updated Runner for those configurations. Image recognition depends on the selected model and the CLI's support for forwarding image content.

Validation

  • Passed the 10-engine routing matrix with a synthetic PNG, including the platform and workspace MCP catalogs, attachment scoping and the large-stdin boundary check.
  • Ran real CodeBuddy, Codex, Claude Code, OpenCode and Pi CLI binaries against local model gateways and confirmed image content in the first provider request.
  • Ran Kimi Code and the repository-pinned DeepSeek Harness against local model gateways. For both CLIs, reproduced the image filtering before the configuration fix and verified that the next model request contained the MCP image after the fix.
  • Ran the patched OpenCode route against the official glm-5.3-flash endpoint with a synthetic PNG. The model correctly replied green circle.
  • Passed frontend build and all 451 chatbot tests, go vet ./..., the public source check, Gitleaks and diff checks.
  • The full Go suite did not pass cleanly. The first run timed out in httpapi and reported runner and scheduling failures. The runner case passed three isolated rechecks and the longer rerun passed the runner package. The longer rerun reported a scheduling fixture cancellation in httpapi. The scheduling fixtures also produced a cancellation on the unchanged chat baseline.

Cursor, Antigravity and ZCode were verified at LoopForge's MCP boundary. Hosted recognition was exercised with GLM via OpenCode.

const assistantReadImageTool = "workbench_read_image"

func appendAssistantReadImageTool(tools []agentToolDescriptor, session *AgentToolSession) []agentToolDescriptor {
if session.Mode != AgentToolAssistant || len(session.AttachmentIDs) == 0 {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AttachmentIDs also includes PDF and text files, so this adds workbench_read_image to the tool catalog in both platform and workspace mode even on document-only turns. Could we keep the default MCP catalog small by reusing the CLI image paths wherever they work?

CodeBuddy, Codex, Claude Code, OpenCode, and Pi already receive current-turn images natively in this PR. For historical images, we can use the native session history where it contains the image, or the staged path where the CLI can read it. Could we try the same staged PNG through Kimi Code’s built-in ReadMediaFile and the pinned DeepSeek Harness’s built-in read_image with the new model capability settings, then check whether the image reaches the provider? Kimi Code tools · DeepSeek Harness v0.2.2-rc.1 tools

If a CLI still needs the MCP fallback, could we verify that it forwards the MCP image to the model and expose the tool and its prompt hint only for authorized images that need that fallback?

@crispinhu27 crispinhu27 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

初审 #1164(head 6c16219,merge-base d58a85a;之后 main 新增的 afc6e7d 只改 chatbot/docs/scripts,与本 PR 改动文件无交集)。

审查范围:会话图片输入在 Claude / Codex / Agent CLI 三条路径上的改动(assistant.go、assistant_engine_cli.go、assistant_engine_codex.go、assistant_engine_agentcli.go、assistant_conversation_attachments.go、agent_tool_mcp.go、agent_tool_catalog.go、runtime/agent_cli.go、runtime/provider_agentcli.go),结合 runner-login Claude 托管桥(runtime/runner_login_claude.go)和附件物化逻辑。

验证(隔离工作区,不含任何凭据):

  • go build ./... && go vet ./...:head 与 base 均通过。
  • go test ./internal/httpapi ./internal/runtime:head 与 base 失败集合相同(Docker/Chromium 相关用例,本机无 Docker/浏览器,CI 上这些检查均为 success)。head 上一度多出的 TestScheduledCodexCacheWriteUsage / TestScheduledNativeUsageReproduction 经 8 轮交替复跑,base 同样失败 5/8,属既有不稳定用例,与本 PR 无关。
  • workbench_read_image 授权:未授权 ID、../../etc/passwd、授权但 owner 缺失三种输入都返回「不在当前会话的可读范围内」,未越权读取。
  • 已确认问题 1 个,见行内评论([P2] runner-login Claude 下 6 MiB 以上图片使整轮失败)。

本 Review 由自动化 Code Review 任务发布,结论只针对上面列出的验证范围。

Comment on lines +239 to +246
input, err := assistantStreamImageInput(prompt, imagePaths)
if err != nil {
return runtime.CommandSpec{}, err
}
// The invocations above terminate with "--", prompt. The structured
// message replaces the positional prompt while preserving the delimiter.
command.Argv = appendNativeArgs(command.Argv[:len(command.Argv)-1], "--input-format", "stream-json")
command.Stdin = input

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] runner-login Claude 下,6 MiB 以上的图片附件会让整轮对话失败

触发条件:使用 runner 登录的 Claude Code(托管桥 isManagedClaude 路径),本轮用户消息带一张约 6.3 MB 到 10 MB 的 PNG/JPEG(附件上限 workbenchAttachmentLimit 是 10 MB,可以正常上传)。

预期:图片要么随消息内嵌,要么像改动前一样退回到按路径 / workbench_read_image 读取,对话正常继续。
实际:这里把整张图 base64 后放进一行 stream-json 帧写到 stdin。base64 膨胀约 4/3,而 runtime/runner_login_claude.go:96 的 bridge scanner 单行上限是 8 MiB,帧超限后 bridge 报 Claude Code 输入超过限制或读取失败,CLI 收不到用户消息,整轮失败。同一张图在 base(d58a85a)上不走 stdin(stdin=false input-format-stream-json=false),模型按附件路径读取,不会失败,所以是本 PR 引入的回退。按代码,本轮所有图片在同一帧里,几张中等大小的截图加起来超过 8 MiB 也会触发(这一点是读代码得出的,没有单独跑)。

复现(临时测试文件,未提交):

  1. httpapi 包:用 7 MiB PNG 调 prepareAssistantClaudeInvocation(..., ImagePaths: [img], RunnerLogin: true),读出 stdin 第一行:image bytes=7340040 stdin frame bytes=9786885 (bridge limit 8388608),argv 带 --input-format stream-json,provider=claude-code。
  2. runtime 包:把这一帧原样经 runCommand 送进托管 Claude bridge(假 CLI 回显每行长度),对照 1 MiB 帧:
    • control-1MiB: frame=1048656 bytes delivered_to_cli=true
    • pr-image-frame: frame=9786885 bytes delivered_to_cli=false err=Claude Code 输入超过限制或读取失败

修复方向:在内嵌前按 bridge 上限检查编码后的帧大小,超出时不内嵌这张图(或整体退回路径 / workbench_read_image);或者把 bridge scanner 上限提到能容纳 10 MB 附件 base64 后的大小,并配套测试。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants